• Text formats and encoding
• “Text as data” - why use text?
• NLP workflow
• Regular expressions
• “Ask me anything”

• Bits
  • Smallest unit of storage; a 0 or 1
  • With n bits, can store \(2^n\) patterns
• Bytes
  • 8 bits = 1 byte (why 1 byte can store 256 patterns)
  • ``eight bit encoding’’ - used to represent characters, such as represented as = 01000001

• a ``character set" is a list of character with associated numerical representations
• ASCII: the original character set, uses just 7 bits (\(2^7\)) – see http://ergoemacs.org/emacs/unicode_basics.html
• ASCII was later extended, e.g. ISO-8859, using 8 bits (\(2^8\))
• but this became a jungle, with no standards: see http://en.wikipedia.org/wiki/Character_encoding

• Unicode was developed to provide a unique number (a “code point’’) to every known character – even some that are”unknown"
• problem: there are more far code points than fit into 8-bit encodings. Hence there are multiple ways to encode the Unicode code points
• variable-byte encodings use multiple bytes as needed. Advantage is efficiency, since most ASCII and simple extended character sets can use just one byte, and these were set in the Unicode standard to their ASCII and ISO-8859 equivalents
• two most common are UTF-8 and UTF-16, using 8 and 16 bits respectively

• Input texts can be very different
• Many text production software (e.g. MS Office-based products) still tend to use proprietary formats, such as Windows-1252
• Windows tends to use UTF-16, while Mac and other Unix-based platforms use UTF-8
• Your eyes can be deceiving: a client may display gibberish but the encoding might still be as intended
• No easy method of detecting encodings (except in HTML meta-data)

• Many different formats contain text
• How many can you think of?
• What problems are encountered?

• psychological states
• sentiment
• “topics”
• Ideology: “left-right” policy positions
• corruption
• cultural values
• power

• Computed from lexical features, typically
• examples are:
  • frequency analysis
  • readability
  • lexical diversity

• A corpus of State of the Union addresses, given each January by the US President
• Uses a descriptive index of “reading ease”, developed in educational science, known as the Flesch-Kincaid measure
• Widely identified trend that this is greatly increasing over time: SOTU addresses are getting simpler

• Only appears in US presidential State-of-the-Union addresses
• Partly an effect of changing audiences and the mode of delivery: from written to spoken form
• Did not hold true for other forms of political communication, either in the US or comparatively

• Human annotation: Part of the research process
  • Authors
  • Students
  • Crowd workers
• Accompanies the data: Part of the data generation process
  • Star ratings for reviews - most movie sentiment databases (Pang, Lee, and Vaithyanathan 2002)
  • Votes for legislative bills - Thomas, Pang and Lee (2006)

• text is not like cats v. dogs
• sometimes not even cats v. dogs is clear

• Map each word or phrase to a “dictionary” of words, associated with a known “sentiment” or psychological state
• Treats matches within each dictionary as equivalent
• Dictionary choice is important, but the computer does all of the counting – so perfectly reliable
• Examples: Linguistic Inquiry and Word Count, or the General Inquirer

Dictionary object with 1 key entry.
- [posemo]:
- like, like*, :), (:, accept, accepta*, accepted, accepting, accepts, active, …
interests, invigor*, joke*, joking, jolly, joy*, keen*, kidding, 
kind, kindly, kindn*, kiss*, laidback, laugh*, legit, libert*, 
likeab*, liked, likes, liking, livel*, lmao*, lmfao*, lol, love, loved, lovelier, ...

• “polysemy” – multiple meanings. kind has three!
• From State of the Union corpus: 318 matches
  • kind/NOUN – 95%
  • kind (of)/ADVERB – 1%
  • kind/ADJECTIVE – 4%
• These are known as false positives
• Other problem: false negatives (what we miss)
  • Missed: kindliness
  • Also missed: altruistic and magnanimous

• a concise and flexible tool for describing patterns in strings
• the “glob” format (as in the dictionary examples) is a simplified version
• Very good coverage in R for Data Science, chapter on “strings”
• Also: Jeffrey Fried, Mastering Regular Expressions

• simplest: match exact “strings”

library("stringr")
x <- c("apple", "banana", "pear")
str_extract(x, "an")

## [1] NA   "an" NA

• case is an option

bananas <- c("banana", "Banana", "BANANA")
str_detect(bananas, "banana")

## [1]  TRUE FALSE FALSE

str_detect(bananas, regex("banana", ignore_case = TRUE))

## [1] TRUE TRUE TRUE

• . matches any character
• * matches no or more of the preceding character
• + matches one or more of the preceding character
• () allow grouping
• [] defines character classes
• \p{} defines categories

• R package stringr
  • including an excellent tutorial “Regular expressions”
• R package stringi
• base grep() is deprecated but possible to use

• beginning and end of match are built into the pattern
• * means no or any characters
• ? means any single character
• built into quanteda: ?quanteda::valuetype

Dictionary object with 1 key entry.
- [posemo]:
- like, like*, :), (:, accept, accepta*, accepted, accepting, accepts, active, …
interests, invigor*, joke*, joking, jolly, joy*, keen*, kidding, 
kind, kindly, kindn*, kiss*, laidback, laugh*, legit, libert*, 
likeab*, liked, likes, liking, livel*, lmao*, lmfao*, lol, love, loved, lovelier, ...

Take MY429: Quantitative Text Analysis (Lent Term)

Textual data hackathon!